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Abstract — A novel example-based process for automated 
colorization of grayscale images using texture descriptors 
(ACTD) without any human intervention is proposed. By 
analyzing a set of sample color images, coherent regions of 
homogeneous textures are extracted. A multi-channel filtering 
technique is used for texture-based image segmentation. For 
each area of interest, state of the art texture descriptors are 
then computed and stored, along with corresponding color 
information. These texture descriptors and the color 
information are used for colorization of a grayscale image with 
similar textures. Given a grayscale image to be colorized, the 
segmentation and feature extraction processes are repeated. 
The texture descriptors are used to perform Content-Based 
Image Retrieval (CBIR). The colorization process is performed 
by chroma replacement. This research finds numerous 
applications, ranging from classic film restoration and 
enhancement, to adding valuable information into medical and 
satellite imaging, and to enhance the detection of objects from 
x-ray images at the airports. 

Computer vision; Image colorization; Fuzzy C-means 
clustering; Gabor transform; CBIR 

I. INTRODUCTION 

Image colorization has been performed through various 
means since the early 20th century, as a very laborious, time- 
consuming, subjective and painstaking manual. Its main purpose 
is to increase the visual appeal of old black and white 
photographs, motions pictures and illustrations. Current methods 
of image colorization can be classified into two different groups, 
Scribble-based and Example-based. Scribble-based colorization 
techniques require a user to scribble color information onto 
appropriate regions of the grayscale image, which is a time- 
consuming task [ 1] . The color information is then spread through 
the image via various algorithms. Example-based colorization 
techniques automate this process by providing an example image 
from which to extract the color information from [2] [3] [4]. This 
process can save a lot of time and requires little or no user 
interaction. However, the results can vary considerably 
depending on the example image chosen. Most techniques still 
require user input in the form of swatches, and use simple texture 
matching methods [3], While the method suggested by Irony et 
al. [2] used a very robust monochrome texture matching method 
with spatial filtering, they suggested that better results could be 
obtained by using improved spatial coherence descriptors, such 
as the Gabor transform. Several other research papers also 
suggested that better segmentation could be achieved by using 
Gabor filters. 



II. PREVIOUS WORK 

A. Texture-based image segmentation 

Accurately identifying the areas of homogeneous textures in 
an image is a key element of the colorization process. In order 
to effectively segment the image based on texture, Malik and 
Perona [5] suggested that a multi-channel filtering approach 
could be used. Jain and Farrokhnia [6] accomplished this by 
using a bank of two-dimensional Gabor filters. 

B. Gabor Transform 

A two-dimensional Gabor function consists of a sinusoidal 
plane wave of some frequency and orientation, modulated by a 
two-dimensional Gaussian. The convolution of an image with 
a bank of Gabor filters creates a set of filtered images containing 
features that responded to the particulate filter. Jain and 
Farrokhnia [6] suggested that feature extraction can be obtained 
by using a nonlinear sigmoid function, and by calculating the 
average absolute deviation (AAD) for each filtered image. 

C. Clustering and Feature Extraction 

Naotoshi [7] used Gaussian smoothing in order to remove 
the smaller areas, combined with a simple K-means clustering 
algorithm in order to extract regions from filtered images. The 
segmentation results were satisfactory but lacked accuracy. 
Texture-based segmentation was recently improved by Dong 
et al. [8] by using a Fuzzy C-Means (FCM) method to generate 
the index map and palette, and by using the Probabilistic Index 
Map (PIM) model to improve segmentation accuracy. Once a 
proper level of clustering has been obtained, smaller regions 
contained inside of a larger region can be removed. Isolated 
small regions of unique texture cannot be used and can also be 
discarded, and the remaining regions are labeled. Once each 
region is labeled, a sample area for each texture is extracted. 
This is done by finding the largest square that can be fitted into 
each identified contiguous region on unique texture. For each 
texture, the color descriptor (the dominant color as per the color 
histogram) and texture descriptors can be extracted and stored. 

D. Content-Based Image Retrieval 

CBIR, the ability to retrieve images by analyzing their 
content, is another research area that has received a lot of 
attention in recent years. Gabor transforms are also the central 
part of the Homogenous Texture Descriptor (HTD), which 
offers a simple yet powerful method for storing texture 
information for retrieval. HTD is part of the "Multimedia 
Content Descriptor Interface" of the MPEG-7 standard, 
which has been successfully used in the fields of Image 
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Classification and Content-based image retrieval [9]. The 
img(Rummager) [10] image retrieval system has implemented 
several descriptors that can be used to retrieve images in various 
situations. In this research, a new process for automated 
colorization is proposed by combining these techniques in a 
new and innovative way. 

iii. automated colorization using texture descriptors 
(actd) 

The first part of the process is to analyze several sample 
color images. In order to effectively segment the image based 
on texture, the multi-channel filtering approach suggested 
by Malik and Perona [5] was used. Gabor wavelet transforms 
have proven to be a very effective method for achieving 
accurate texture-based image segmentation. A clustering 
algorithm then needs to be used to segment the regions of 
homogeneous texture previously identified and extract a 
representative sample for each texture. The texture 
descriptors and color information can be computed for each 
texture, and stored in a database. The second part of the 
process assumes that a new grayscale image needs to be 
colorized based on the texture and color information present 
in the database. The segmentation and feature extraction 
process takes place as previously described. For each texture 
identified, the texture descriptors are computed, and used to 
locate the best matching texture present in the sample 
database. Once the best matching texture is identified, the 
corresponding color information is extracted, and applied to 
the segmented region of the grayscale image. Figure 1 shows 
an overview of the ACTD process used in this research. 
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Figure JL Architecture for ACTD 
A. Texture-based Image Segmentation 

In order to obtain a good texture -based segmentation of an 
image, the multi-channel filtering approach suggested by Malik 
and Perona [5] was used. This was accomplished by using a 
bank of two-dimensional Gabor function consisting of a 
sinusoidal plane wave of a given frequency and orientation, 
modulated by a two-dimensional Gaussian envelope. The Gabor 
filters were obtained using equation (1) 

g(x,y;A, <j,y) = exp l-y * + ^ y J cos \2tz ^) (1) 

x ' — x cos + y sin (2) 

y' — —x sin + y cos (3) 

Where 6 is the width of the Gaussian envelope, a is the spatial 
aspect ratio, e is the wavelength of the filter, and e is its 



orientation. Jain and Farrokhnia [6] achieved good feature 
extraction results by using a nonlinear sigmoid function and by 
calculating the average absolute deviation (AAD) for each 
filtered image. Gaussian smoothing was used in order to remove 
the smaller areas. In this research, smoothing was performed 
by calculating the mean values of the gray-level intensities 
between neighboring pixels. The Gabor response values for 
each pixel were added, and normalized. 

B. Clustering 

Jain and Farrokhnia used the K-means clustering algorithm 
to create and label the texture regions. The Fuzzy C-means 
clustering (FCM) algorithm can improve this process by 
allowing the concept of partial membership, in which an image 
pixel can belong to multiple clusters. This "soft" clustering 
allows for a more precise computation of the cluster 
membership. 

C. Modified Fuzzy C-means clustering with "G k . factor" 

In order to improve the tolerance to noise of the Fuzzy C- 
means clustering algorithm, Krinidis and Chatzis [11] have 
proposed a new method by introducing the novel G w factor. 
The purpose of this algorithm is to adjust the fuzzy 
membership of each pixel by adding local information from 
the membership of neighboring pixels. It is obtained by using 
a sliding window of predefined dimensions used to compute 
the G ki factor. The G ki factor is calculated by using: 

,mi i2 



G 



(8) 



j * i J 
Where the i* pixel is the center of the local window, the j* 
pixel in the window around the i th pixel (N), k is the reference 
cluster, d is the spatial Euclidian distance between pixels i and 
j . \i k . is the degree of membership of the j th pixel in the k ,h cluster, 
v k is the prototype of the center of cluster k, m is a fuzziness 
factor (a value > 1 ), x is the i th pixel in N, |x. - vj is the Euclidean 
distance between x. and v k This algorithm is minimized by using 
equation (4): 

ZfUE^il/^l^ - i^l 2 + G ki \ (4) 
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D. Feature Extraction 
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The contiguous region on unique texture enhanced by the 
Gabor filters and identified by the segmentation algorithm are 
then isolated and extracted. Blob filtering is used in order to 
remove the smaller clustered areas. The center of gravity of 
each blob is used to extract a sample image representative of 
that particular texture. The texture image is stored both in color 
for a reference to the color component (chroma) of the texture 
and in grayscale for texture matching. 

E. Grayscale image processing 

The segmentation and feature extraction processes are then 
repeated for the new grayscale image to be colorized. 
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F. Texture Matching 

Compact Composite Descriptors (CCD) is a recently 
proposed set of descriptors combining several features 
descriptors. These Descriptors are part of the new Visual 
Multimedia Content Description Scheme (VICODEs), which 
is proposing a set of specialized descriptors tuned for different 
types of images. Testing of these descriptors was performed 
using the img(Rummager) [10] application, using the MPEG-7 
Edge Histogram Descriptor (EHD), the VICODEs (CCD) Fuzzy 
Spatial Based Scalable Composite Descriptor (BTDH) [12], and 
the VICODEs (CCD) Auto Descriptor Selector (ADS). 

G. Grayscale Image Colorization 

Once the proper segmentation is obtained and the proper 
colors are identified, placing the color in the proper areas of the 
test image can be achieved in the YCbCr color space. The 
advantage of this color space is that it separates the luminance 
(Y) component from the color components (Cb and Cr are the 
blue-difference and red-difference chroma components). Thus, 
it is possible to replace the chroma components while preserving 
the luminance information of the image. In this research, the 
chroma component of the test image is replaced with the colors 
corresponding to the matching textures from the library of 
images. 

IV EXPERIMENTAL RESULTS 

Figure 4 shows the various stages of the colorization of a 
test image through the ACTD process. 




Figure 4. (a) Grayscale image, (b) ) Mean smoothing of the sum of 
Gabor responses, (c) Segmented regions, (d) Colorized image 

V.CONCLUSION AND FUTURE WORK 

A new and innovative method for automating example -based 
colorization process is implemented. This process combines 
several techniques from Digital Image Processing in order to 
improve the automation of the colorization process. This includes 
Gabor-based image segmentation combined with improved 
fuzzy C-means clustering, extraction and storage of the Texture 
and Color Descriptors, and a texture -based color retrieval 
technique. Colorization results in general are subjective and not 
easily quantifiable. Results obtained using this method are 
dependent on the number of textures present in the database 
and on the ability to find a suitable color match. 



However, reasonably accurate results were obtained with a 
limited number of sample images. Several of the steps require 
custom parameters that can be tuned for different types of 
images, such as the Gabor wavelength and orientation, and the 
number of clusters. Since the textures extracted from each 
sample image have no preset size or shape, the Texture retrieval 
methods need to be improved for scale and rotation invariance, 
which would lead to a better accuracy rate. This method could 
also be enhanced to store more complete color descriptors in 
order to accommodate more complex textures containing 
multiple colors. These improvements would most likely improve 
the accuracy of the colorized images, the testing conducted as 
part of this research proved that the ability to combine these 
techniques in order to automatically colorize grayscale images 
is a viable option. 
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